

agenda for Ed Tech. 


Research Windows 


Putting the Pieces Together 


Explore the idea and purpose 
of developing a research 


Leaders Sharing 


G len Bull’s call for a research 
agenda for one-to-many 
computing (see p. 42) raises 
a larger question: what is the status of 
a research agenda within education 
technology? 

Let’s consider first what a “research agenda” is in 
general. The Merriam-Webster dictionary defines re- 
search as “studious inquiry or examination.” (There’s 
more to the definition than that, but this is a good 
starting point.) It further defines agenda as “a list 
or outline of things to be considered 
or done.” So if we want a research 
agenda in Ed Tech, we need to make 
a list of, say, options for or types of 
studious inquiry or examination of 
education technology. 

By Robert Kadel 


Subject: Ed Tech research 


Standards: NETS*A II (http://www. 
iste.org/nets/) 


Supplement: http://www.iste.org/ll/ 


We are certainly not alone in 
considering this issue. Try going 
to Google and typing the words 
“research agenda” into the search 
box. Hit Enter, and you’ll get the 

usual millions of hits that 
Google can find these 
days. Looking at the 
first page, you’ll see that 
research agendas have 
been proposed in dis- 
ability data, occupational 
research, information architecture, 
Parkinson’s disease, and music educa- 
tion, among others. 

So why not us? Certainly one 
of the most ridiculed but 
common reasons for do- 
ing something in research 
is because “everybody else 
is doing it.” But in this 
case, what everybody else 
is doing seems to be on the 
right track. 

I recently wrote about research on 
teacher attitudes regarding technol- 


ogy integration into instruction (L&L 
February 2005, pp. 34-35, 47), and 
in this article I mentioned that no 
one researcher can answer a broad 
research question. It takes a com- 
munity of scholars who all contribute 
their valuable research — pieces to a 
large puzzle that is the answer to that 
broad research question. 

So a research agenda can help 
guide that community of scholars. A 
researcher in any of these other, high- 
ly varied fields benefits from having a 
specific vision or a set of goals for the 
types of research that will help his or 
her field advance more efficiently and 
effectively. Furthermore, a research 
agenda helps to outline particular steps to be tak- 
en and particular methodologies to be used. (Yes, 

I say “methodologies” — plural — and I’ll expand on 
this shortly.) 

John Cradler’s Research Windows column 
from the April 2003 issue of L&L 
(pp. 54-57) summarizes CARET’S 
(Center for Applied Research in 
Educational Technology, http: //caret. 


28 


Learning & Leading with Technology Volume 32 Number 8 

Copyright © 2005, ISTE (International Society forTechnology in Education), 1 .800.336.5 1 9 1 ( U.S. & Canada) or 1 .54 1 .302.3777 (Int'l), iste@iste.org, www.iste.org.AII rights reserved. 










% 

d 




iste.org) opinion, based on reviewing 
more than 1,200 studies, that there is 
a whole host of problems in how Ed 
Tech research has been conducted. 

Beyond that list, though, Cradler and 
CARET identify nine goals that must 
be accomplished within Ed Tech re- 
search to help educators plan and use 
technology effectively. For example, 
the research community must ensure 
that assessments are directly related to and valid 
measures of curriculum standards, both those pro- 
vided by the state and those addressed by the soft- 
ware being used. This seems like a given, but you’d 
be surprised (or perhaps you wouldn’t) how many 
research projects don’t actually address the issues 
they’re supposed to be studying — neither by ask- 
ing the right questions nor by using a 
methodology appropriate to what is 
being studied. ( Editor’s note: You can 
find Cradler’s article in L&L’s archives 
at http://www.iste.org/ll/.) 

And yes, despite the buzz that 
emanates from Washington, D.C., 
a number of methodologies are 
appropriate to Ed Tech (and all 


educational) research besides experi- 
mental/ quasi-experimental design. 
Depending on how far a research 
question has progressed through the 
community of scholars, case studies, 
interviews, focus groups, surveys, and 
so on are needed. All of these can il- 
luminate central questions that can 
then be addressed by randomized, 
controlled trials in an experimental 
design. Anything less than choosing 
the methodology appropriate to the 
question at hand is irrespon 
sible. And, in my opinion, 
there should be adequate 
state and federal funding 
for any of these method- 
ologies if it is suitable to 
the question at hand. 

In volume 36, number 1 
of the Journal of Research on Technol- 
ogy in Education (pp. 60-76), M. D. 
Roblyer and Gerald A. Knezek move 
us that much closer to a national Ed 
Tech research agenda by proposing 
four topics that should be central 
to what we study in this field. The 
first, “research to establish a relative 
advantage,” focuses on establishing 
which technology-based methods of 
instruction can provide unique and 
consistent benefits. That is, we need 
to know what types of tech-based in- 
struction provide a one-of- 
a-kind benefit not found 
elsewhere, and that these 
benefits are seen across 
a range of different stu- 
dent populations. ( Edi- 
tor’s note: Read Roblyer 
and Knezek’s article in the 
online supplement.) 

The second topic, “research to 
improve technology implementation 
methods,” seems self-explanatory, 
but allow me to clarify. A number of 
types of technology are already being 
used in schools or have the potential 
to be used, from word processors and 
graphing calculators to electronic 


whiteboards and highly sophisticated 
curriculum management systems. 
Research is needed to determine the 
most effective methods for imple- 
menting these vast and varied re- 
sources. (In addition, don’t forget, 
this research needs to be unique and 
consistent; see above.) 

Roblyer’s and Knezek’s third topic? 
“Research to monitor technology’s 
impact on important societal goals.” 
Because of my sociology background, 
this one is my favorite. For example, 
we need to know how technology in education 
is helping to close the gap in achievement scores 
between students from low-income households 
and those from high-income households. 

Finally, Roblyer and Knezek propose that we 
conduct “studies that monitor and report on 
current technology uses to help us 
shape desired directions.” The key 
term here is desired directions — we 
need research that looks at how we 
currently use technology to help us 
understand the ways we should use 
it in the future. What do we want to 
accomplish in educational technol- 
ogy? Well, like so many other things 
in life, it is easiest to determine where 
we want to go by first looking at 
where we are. 

Bull’s article this month meets 
a number of the criteria discussed 
above, and he should be commended 
for it. Now it is time for the rest of us 
to throw our hats into the ring — we 
are all a part of this community of 
scholars and we must work together 
to bring about the research, and ulti- 
mately the changes, that we all wish 
to see in our schools. 


Robert Kadel is the founder 
and a general partner of Kadel 
Research Consulting, LLC, 
located in Hyde Park, Vermont. 
His firm focuses on the evalua- 
V tion of educational programs in 
K technology, school reform, and 

community involvement. Robert serves as volun- 
teer editor of L&L’s Research Windows column. 
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Abstract 

Research to indicate unique pedagogical benefits of technology in education flourished until 
the mid-1980s, when criticism of the premises and methodologies of studies made educators 
question the usefulness of so-called “media research. ” Currently, increasing costs of main- 
taining technology infrastructure and evidence of low use of technology by teachers has 
generated new urgency for research to help provide a rationale for specific uses of technology 
to support teaching and learning. The new National Educational Technology Plan cur- 
rently under development by the U.S. Department of Education provides an ideal opportu- 
nity for articulating a plan for meaningful research. This article describes topics and issues 
on which new research should focus and offers guidelines for research methods to address 
previous criticisms and provide results to guide future use of technology in education. (Key- 
words: educational technology, instructional technology, educational technology research, 
research methods, research agenda, technology impact studies.) 


INTRODUCTION 

Three assumptions educators often make about research on technology-based 
(i.e., computer-based and online distance learning) methods are that: (a) it has a 
commonly-held, theory-based agenda, (b) findings provide convincing evidence 
about the unique ways modern technologies enhance achievement and motiva- 
tion, and (c) these findings shape practice in the field. The last thirty years of 
educational technology research notwithstanding, none of these assumptions 
are currently true. The authors propose that current conditions in education 
and society offer the combination of urgent need and unique opportunity to 
make all of these assumptions realities. A review and analysis of the history of 
recent research on electronic technologies in education reveals key issues and 
problems that must be addressed as we contemplate a research agenda for the 
next era in technology use in education. 

BACKGROUND ON THE EDUCATIONAL TECHNOLOGY 
RESEARCH AGENDA 

As Richey (1997) pointed out, agenda-building in a field as prone to change 
as educational technology is a “complex process . . . often dependent on far 
more than whether or not an idea is worthwhile” (p. 8). Yet the importance of 
what she calls a “systemic theory agenda” cannot be overstated. It “captures the 
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intellectual climate of a field . . . identifies those issues currently receiving major 
attention ... is supported by influential scholars of the day ... is heeded by ex- 
pert practitioners . . . influences the selection of articles for publication in jour- 
nals of our field (and) paper proposals accepted for presentation at our national 
conferences . . . and shapes university curricula which prepare and update the 
professional in our field” (p. 7). 

Until the early 1980s, the theoretical foundations of educational technology re- 
search were not widely debated and the research agenda based on them seemed 
comparatively clear. The theory base was primarily cognitive-behaviorist, and re- 
search into technology-based methods reflected this view. In addition to numerous 
evaluations of computer-based and distance education applications and products, 
there were many experimental and quasi-experimental “media comparison” studies 
comparing technology-based with non-technology-based methods to establish 
which one was better for teaching a topic or skill. A thriving meta-analysis culture 
had the goal of identifying trends in findings across these studies. The question 
common to all these enterprises was fundamentally a cognitive-behaviorist one: 
“How effective is a technology in helping learners acquire desired behaviors 
(knowledge and skills demonstrated by an observed and measured performance)?” 

Attacks on this solid bastion of traditional thinking came from two direc- 
tions, one theoretical/pedagogical and one methodological. Working together, 
they constituted an assault that effectively called a halt to media comparison 
studies and meta-analyses and left the field without a clear philosophical frame- 
work or workable research agenda. 

The first attack came from educators, psychologists, and researchers who es- 
poused a constructivist perspective on learning. They said that the aim of education 
was not to transmit learned behaviors from an expert source (an instructor or a 
package of instruction) to students. Instead, it was to create environments in which 
students generated their own knowledge with appropriate assistance or “scaffold- 
ing” from teachers. Seymour Papert (1980) was an early and outspoken representa- 
tive of this perspective, which he based primarily on the theories of Piaget. Later, 
the Cognition and Technology Group at Vanderbilt (1991, 1993), calling on work 
by Vygotsky in cognitive scaffolding and Brown, Collins, and Duguid (1989) in 
situated cognition and cognitive apprenticeships, became a major hub of construc- 
tivist research and development. These theorists and researchers turned the focus 
away from the impact of the technology being used to the impact of “anchored in- 
struction,” which technology could support. 

At about the same time, Clark’s (1983, 1985, 1991) criticism of the research 
methods used in studying technology-based approaches began a parallel attack 
on the traditional research agenda. Clark said that “media are delivery vehicles 
for instruction and do not influence learning” (1983, p. 453). He based this on 
analyses of past media research, which he found to have uncontrolled effects 
due to methods and content. For example, he found that two methods being 
compared often had different teachers. He concluded that any effects attributed 
to a technology were, in fact, caused by the methods used by the teacher or 
other variations in the content of the treatments. Based on these issues, he 
called for a moratorium on media research. 
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Kozma (1991) tried to address both theoretical and methodological problems 
by agreeing partially with Clark. Kozma said that, as long as media were viewed 
strictly as delivery systems, there would be no discernible difference among 
treatments that was not due to confounding factors. Instead, he proposed a 
theoretical framework that “presents an image of the learner actively collaborat- 
ing with the medium to construct knowledge” (p. 179). He felt that the “sym- 
bol systems” and “processing capabilities” unique to each medium could make 
an identifiable and useful contribution to this collaboration. While advocating 
more research to clarify these unique contributions, he left the guidelines for 
this development and research up to the “creativity of designers” (p. 206) and 
those who would study their products. 

Kozma’s proposal was helpful in redirecting the focus on studying the contri- 
butions of the technology-based method, rather than the technology or “me- 
dium” itself. Subsequent research such as the well-regarded Technology Counts 
study (Archer, 1998) seemed to be heeding this line of reasoning, asserting that 
it was not how much teachers used computers but how they used them that 
made a difference in students’ achievement. 

At the same time, pragmatism born of experience with students and class- 
rooms dictates that we modify pure constructivism as a theory basis for a new 
research agenda. In his call for a “post-modern agenda for instructional technol- 
ogy (IT),” Soloman’s (2000) insights are useful. He maintained that “The 
philosophical core of post-modern IT is a belief in pluralism, which can be de- 
scribed as respect for difference and resistance to single explanations . . . plural- 
ism in our field posits that there is no single best model or theory of learning” 
(p. 15). He suggests that the sheer complexity of the learning context calls for a 
closer look at the combination of method, learner, and environmental variables 
that contributes to successful learning. 

In Soloman’s “post-modern” view, both “scientific” and “critical” methods 
must be used to study this complex picture. Science is “problem-driven and 
goal-based” (p. 16), and looks at whether objectives have been achieved. Criti- 
cism looks at the details that could account for the outcomes. Like many cur- 
rent theorists, Soloman clearly emphasizes the constructivist view of learning as 
“knowledge (being) created rather than found” (p. 16), but his 
acknowledgement of the need for scientific study allows for the statement of 
specific problems to investigate and measurable observations of changes in be- 
haviors: two conditions the cognitive-behaviorists have always said should be 
the basis for establishing that learning has indeed occurred. 

The merging of critical features of these heretofore competing theories sets 
the stage for a research agenda to study the benefits of technology-based meth- 
ods as integral components of solutions to instructional problems, rather than 
as “delivery media.” Although we recognize that this proposal will not (and 
probably should not) silence the ongoing debate about the principles and prac- 
tices of cognitive-behaviorist versus constructivist approaches to learning, we do 
feel that some detente in the theory debates is essential if we are to progress to 
the work that must follow them: crafting and studying technology applications 
that work in actual instructional settings. 
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THE NEED AND OPPORTUNITY FOR A “NEW MILLENNIUM” 
RESEARCH AGENDA 

The need for a new research agenda has never been more urgent. Recent 
projects under the U.S. Department of Education’s (DOE’s) Preparing 
Tomorrow’s Teachers to use Technology (PT 3 ) initiative have demonstrated 
clearly that the process of integrating technology effectively into education 
comes with a high price tag. Educators and policymakers look to educa- 
tional technology research to help supply a solid rationale for why these 
funds are well spent (Ringstaff & Kelley, 2002). This is increasingly impor- 
tant in light of recent criticisms of the continuing investment required for 
technology infrastructure and the inconsistent impact and low usage by 
teachers, regardless of their increased training and access to resources (Cu- 
ban, Kirkpatrick, & Peck, 2001). 

Cradler (2003) stresses that the recently enacted No Child Left Behind 
(NCLB) Act (2002), which will be influential in the next few years, requires 
that learning resources such as electronic technologies “be closely aligned to 
state academic standards and that research document the extent to which (they) 
result in student achievement gains as measured by standards-based assessment” 
(p. 35). However, in his review of “best evidence” studies posted by the Center 
for Applied Research in Technology (CARET), a funded project of the Interna- 
tional Society for Technology in Education (ISTE), Cradler found that elec- 
tronic learning resources aligned to less than 40% of the California Content 
Standards for the major content areas. He also found a “shortage of formal con- 
trolled research” (p. 56). Even with experimental studies that reported statistical 
significance, there was “very little information on the educational significance 
or the implications for teaching or learning” (p. 56). 

In addition to the NCLB requirements, the DOE is currently calling for in- 
put from educators to create a new National Educational Technology Plan 
(www.nationaledtechplan.org). As Richey (1997) observed, another component 
of an agenda-setting process is “support of credible and authoritative people in 
the field . . . The larger the base of support, the more likely a topic is to become 
part of the discipline’s systemic agenda” (p. 9). The DOE’s National Technology 
Plan could be the single most influential force on the educational technology 
research agenda for the future. It is crucial that the important topics to be stud- 
ied are articulated now and offered as an integral part of the plan. 

FOCUS AND TOPICS FOR A FUTURE TECHNOLOGY RESEARCH 
AGENDA 

Recommended Focus: Providing a Rationale for Technology Use 

In light of the pressure from policymakers and from within the field itself, the 
primary task of the new research agenda seems clear. Future research must ad- 
dress squarely the question of why teachers should use technology-based meth- 
ods. The emerging theory base demands that studies look at technologies not as 
delivery systems, but as components of solutions to educational problems, and 
that research questions be stated in a way that the contributions of methods can 
be examined and tested. 
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Roblyer and Schrum (2003) say that rationale-building research should con- 
centrate on the “relative advantage” that technology-based methods and materi- 
als offer over other methods and materials. This concept is based on Rogers’ 
(1995) diffusion of innovations research that shows potential adopters of a tech- 
nology are more likely to accept it if they see clear benefits or “relative advan- 
tage” of new methods over old ones. 

In his talks to groups of educators around the country, former PT 3 Program 
Director Tom Carroll (2000) emphasized that some technologies in common 
use have obvious relative advantage that needs no research to confirm it. For ex- 
ample, when computer simulations save schools the costs of consumable mate- 
rials or allow multiple simulated experiments that are not feasible using actual 
environments, relative advantage should be readily apparent. However, even 
methods with self-evident benefits are in need of explication to realize their po- 
tential in practice. In addition, other methods have been hypothesized to have 
unique instructional benefits and some have popular support, but research to 
confirm these benefits has been inconsistent or done with inadequate controls 
or numbers of subjects to make findings useful to others. 

Recommended Topics: Important Issues to Address 

This dual view of technologies as both an already- accepted part of some areas of 
educational practice and as a potential influence in others requires a multifaceted 
approach to providing research that can guide both policy and practice. We pro- 
pose that the studies that would be most helpful are those intended to indicate: (a) 
that certain technology-based methods have the potential for unique and fairly 
consistent benefits in response to certain kinds of educational problems; (b) there 
are ways to implement technology-based methods already in widespread use (e.g., 
word processing, distance learning) that lead to increases in their impact on 
achievement, retention, and student satisfaction; (c) progress we are making to 
meet some of educations most important goals for technology (e.g., decreasing the 
Digital Divide); and (d) how we can shape and direct our technology resources to 
better accomplish their intended purpose. These purposes and examples of studies 
that address them are described briefly here. 

Research to establish relative advantage. In their review of findings to date on 
the impact of technology-based methods on achievement, Ringstaff and Kelley 
(2002) find that research has provided considerable evidence that directed 
methods such as using software tutorials for self-paced instruction and drill and 
practice for enhancing retention of learned skills can significantly raise scores on 
standardized tests. However, even this evidence is not as helpful as it could be, 
because studies tend to focus on “a specific software or technology, severely lim- 
iting the generalizability of their results . . . and fail to take into account the 
larger classroom, school, and district context in which the technology was used” 
(p. 6). Ringstaff and Kelley also acknowledge that evidence of benefits fre- 
quently proposed for more constructivist, inquiry-based uses is also “far from 
conclusive” (p. 7). Research is needed to indicate that the following often-pro- 
posed benefits are possible: 
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• The graphics and interaction features of modern visual technologies have 
unique qualities to increase and speed comprehension of unfamiliar or ab- 
stract topics by clarifying their underlying concepts. 

• Visual examples provided by problem-solving software, simulations, and 
video scenarios prevent inert knowledge by linking the concepts to real-world 
applications. (Brown, Collins, and Duguid, 1989, describe inert knowledge 
as skills that students learn but never use because they cannot see where or 
how they apply to problems they encounter in real-life situations.) 

• Online collaboration projects encourage greater appreciation of diverse cul- 
tures and improve reading and writing skills. 

Studies that address these kinds of claims are beginning to appear. For ex- 
ample, Doty, Popplewell, and Byers (2002) looked at whether the interactive 
qualities of CD-ROM storybooks could result in greater comprehension for 
students than reading in a static format, and Funkhouser (2002-2003) explored 
the impact on achievement of constructivist instruction made possible by the 
unique capabilities of geometry software, as opposed to the traditional instruc- 
tion that non-electronic materials are capable of accomplishing. 

Research to improve technology implementation methods. A casual reading of 
any professional journal or education periodical reveals that many technology- 
based methods are already in widespread use (e.g., word processing, distance 
learning). Research is urgently needed to help shape the ways in which these 
technologies are being implemented to increase their impact on established 
measures of education quality: achievement, retention, and student satisfaction. 
Even accepted “older” technology-based solutions such as using word processing 
for composition writing have not been sufficiently studied in practice. In re- 
viewing evidence on how people grade word-processed vs. handwritten prod- 
ucts, Roblyer (1997) found that reviewers who had not been trained to guard 
against a bias tended to give word-processed products significantly lower scores. 
An example of recent research into commonly-used online methods is 
Overbaugh’s (2002) study of how electronic mailing lists and bulletin boards 
could be implemented more effectively to take advantage of unique features 
while preventing unintended outcomes. Findings from such studies have im- 
portant implications for future practice. 

Research to monitor technology’s impact on important societal goals. Although 
some technology uses in education require more research to justify them, access 
to technology is acknowledged as being an essential part of a quality education 
for citizens in todays Information Society. We also need to know that increased 
access to technology is resulting as expected in greater learning opportunities 
and skills for all students. Studies are needed to monitor our progress in accom- 
plishing the following kinds of important goals: 

• Technologies are helping underserved students gain increased access to learn- 
ing opportunities. 

• Ready access to unique information sources and populations is enhancing 
critically important information literacy skills. 
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• More use of visual technologies is increasing critically important visual lit- 
eracy skills. 

Useful studies in this area are often surveys such as those by Hoffman and 
Novak (1998) to capture the impact of race on Internet access and computer 
usage. Others are studies such as the one by Chisholm, Carey, and Hernandez 
(2002), who reviewed access to and uses of information technologies by stu- 
dents of various ethnic backgrounds. 

Studies that monitor arid report on current technology uses to help us shape 
desired directions. Finally, research is needed to shape the characteristics of our 
newer technology resources and guide their uses in education. Mioduser, 
Nachmias, Lahav, and Oren (2000) examined the characteristics and usefulness 
of educational Web sites to determine how they should be better developed and 
implemented in the future. Harris and Jones (1999) studied communicadon 
patterns of online mentoring to determine how this kind of activity could be 
better implemented to increase impact. Research is needed to help shape appro- 
priate uses of these and many other technologies (e.g., handheld computers) be- 
ginning to appear with increasing frequency in schools. 

CRITICISMS OF PAST RESEARCH AND GUIDELINES FOR FUTURE 
METHODS 

Clarks (1983, 1985, 1991) criticisms of the focus and methods of technology re- 
search are not the only negative evaluations the field has received. The usefulness of 
findings in this area has long been compromised by a reliance on funding from the 
very groups who, as Clark pointed out, would profit most from positive findings: 
hardware and software companies. There is also consistent evidence of what can 
only be characterized as basic design and methodological flaws in many studies. 
Much of educational research in general is often the target of the latter criticism 
(Kaestle, 1993). However, technology research is perhaps in a rather more tenuous 
position than many other areas of education, in that it has long been in need of a 
rationale sufficient to justify its high costs and complexity. Guidelines for future re- 
search are needed that recognize these flaws and offer means of remedying them. 

Guidelines for Research Hypotheses 

Clark (1983) cited design flaws common to individual studies (e.g., different 
teachers for experimental and control groups) and the meta-analyses that sum- 
marize findings across them (e.g., tendency for publications to accept primarily 
studies that report significant, rather than non-significant, differences). Kozma 
(1991) and others acknowledge that a research agenda based on showing the 
“overall impact of one medium over another across learners” (p. 204) will not 
guide us into the new millennium of technology in education. However, Clarks 
criticisms and Kozma’s acknowledgement expose a far deeper failing inherent in 
much of educational research: flaws in the ways hypotheses are (and, often, are 
not) thought about and posed. 

It is far from clear in many studies that researchers have considered carefully 
why given methods are being proposed for testing. It is axiomatic that educa- 
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tional research should help address heretofore unresolved or poorly resolved 
educational problem(s), and that hypotheses should be derived from a theory 
base that leads logically to the hypothesis as a solution to the problems (s) and 
makes it clear why it should be tested. 

Guidelines for Design Methods 

In his review of recent “best evidence” technology research, Cradler (2003) al- 
ludes to several kinds of design flaws, some of which indicate underlying prob- 
lems with hypotheses. He says that it is often “not clear what specific research 
and evaluation questions the study is attempting to answer” (p. 56) or its “edu- 
cational significance or ... implications for teaching and learning” (p. 56). 

Other design flaws he cites include: small sample sizes that reduce statistical sig- 
nificance; measures of impact unrelated to objectives of the materials; measures 
of impact that are not objective and/or validated; descriptive, non-quantitative 
methods only with no control or comparison groups; and self-report data unac- 
companied by corroborating data. 

Not only must future research refrain from simplistic comparisons of materi- 
als cited by Clark, it must begin to focus on the unique qualities of technologies 
to make possible given methods. If a study proposes that a technology-based 
method is superior, it should be hypothesized to be superior because it has 
unique characteristics to solve a significant, unresolved problem such as low 
reading comprehension (as addressed by the Doty, Popplewell, and Byers, 2002, 
study) or students’ persistent difficulties with geometry concepts, as addressed 
by Funkhouser (2002-2003). Studies must offer evidence — both in design and 
in data analysis — that the symbol systems and/or processing capabilities of a 
technology make possible an instructional method or learning environment 
that would not be possible through another (i.e., non-technology) method. 

Because the NCLB Act will have a long-term impact on all future educational 
research methods, it is important to acknowledge the criticism of past methods 
implicit in its recommendations. It is clear that so-called “scientifically-based re- 
search (SRB) methods” mentioned more than 100 times in the NCLB Act re- 
flects a pervasive impatience with the failure of past research to provide objec- 
tive, helpful evidence to guide practice. Cradler (2003) recommends that, in 
addition to using multiple data sources and validated measures, studies should 
set up control groups whenever possible and use valid comparison groups even 
if true controls are not possible. 

It seems important to join those who, in the aftermath of the NCLB Act, have 
asserted that the failure of past research cannot be remedied solely by using experi- 
mental and/or quasi-experimental methods. Berliner (2002) reminds us that one of 
the largest and best-funded replication studies ever, one of different instructional 
models of early childhood education, found that variation was larger across differ- 
ent sites implementing the same program than it was across different programs. 

“No program could produce consistent results across sites. Each local context was 
different, requiring differences in programs, personnel, teaching methods, budgets, 
leadership, and kinds of community support” (p. 19). Thus, he says, “good science” 
requires not only trying to capture valid and reliable evidence of improvement, but 
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also attempting to explain the complex set of conditions that led to improvement. 
As Soloman (2000) recommended, critical analysis methods must join “scientific” 
ones to help us understand what works, when it works, and why. 

Reporting Guidelines 

Cradler’s (2003) review of past research found many reporting flaws that are 
just as problematic as design ones, because they limit both interpretation of 
findings and studies designed to replicate them. Reporting flaws include: insuf- 
ficient information on the intervention and/or technology and other resources 
being implemented and the characteristics of large samples that allow disaggre- 
gation of important variables; and a disconnect between the conditions under 
which technologies are used and their conclusions, or between the conclusions 
and the data cited to support them. 

Reports of research studies must address these frequently-cited shortcomings. 
They must be more comprehensive and informative about the methods and 
materials used, conditions under which studies take place, data sources and in- 
struments, and subjects being studied; and they must emphasize coherence be- 
tween their methods, findings, and conclusions. 

CONCLUSION: IMMEDIATE AND LONG-TERM PRIORITIES 

To explore the degree to which current research may already be addressing the 
four kinds of needs cited here, we analyzed the purposes of articles in the 1 999— 
2003 issues of the Journal of Research on Technology in Education (JRTE). Find- 
ings from this analysis (Table 1) revealed that most articles focus on evaluations 
of local programs or on methods of furthering technology integration or im- 
proving teacher training. Our analysis provides further evidence of what Cradler 
has already found. Relatively few studies are being done for purposes that 
would help justify the costs of integrating technology-based methods. 

As Table 1 indicates, most published research articles describe methods of get- 
ting students or faculty more involved with a technology (e.g., the Internet) or 
how to structure training and other conditions to get them more interested in 
using technology in general. From the perspective of the NCLB Act, this kind 
of research begs the question of “Why should we have students or faculty use 
technology more?” 

It is to be expected the National Education Technology Plan now under de- 
velopment will provide considerable and immediate impetus to focus on these 
and / or similar topics and to improve the quality and usefulness of research. 
Hopefully, this plan also will make it easier to obtain funding from sources 
other than those that could be perceived as having a self-interested agenda. 

However, groups Richey (1997) describes as the base of support for a technol- 
ogy research agenda will have the most direct, long-term impact on how well 
studies that support a new agenda are carried out: influential scholars and ex- 
pert practitioners who guide dissertation research, select articles for publication 
in journals of our field and paper proposals for presentation at our national 
conferences, and who shape the curricula of teacher preparation programs. 

These groups will help shape the next millennium of technology in education as 
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Table 1: An Analysis of Article Types in JRTE from 1999—2003 

Number of JRTE Articles 


Types of Articles 

1999-2003 

Type #1: Studies or reviews of unique benefits 
of various technology-based methods compared 
with non-technology-based methods 

1 1 (8%) 

Type #2: Studies on ways to increase impact 
of technologies already in common use 

2 (1%) 

Type #3: Studies on trends in usage related 
to achieving important societal goals 

8 (6%) 

Type #4: Studies that monitor/report current 
technology uses to help shape desired directions 

5 (4%) 

Type #5: Other studies, e.g., evaluations or 
descriptions of programs, implementation 
methods, or usage characteristics; types, extent, 
or perceptions of integration 

87 (64%) 

Type #6: Theoretical articles and position papers 

23 (17%) 


they view proposed studies or reports of studies through the lens of whether 
they address one of these key, unmet needs and meet design and reporting re- 
quirements. It is they who will determine that research findings are meaningful 
and helpful to those who look to them for guidance. 
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Answering the Call: A Response to 
Roblyer and Knezek 


Neal Strudler 

University of Nevada, Las Vegas 


First, I’d like to express my gratitude for the opportunity to respond to the 
Roblyer and Knezek article. I believe that it raises some critical issues about for- 
mulating a much-needed research agenda for educational technology and I ap- 
preciate being part of the conversation. 

Let me begin with the authors’ assessment of current educational technology re- 
search — that it tends to lack sufficient theory, fails to provide adequate evidence 
about how modern technologies enhance achievement and motivation, and fails to 
adequately shape practice in the field. That’s a strong statement, but unfortunately, 

I tend to agree. I do appreciate, though, the positive manner in which theses issues 
were framed, focusing on the challenges and opportunities that we face. It should 
be noted that designing and implementing technology-based learning activities in 
schools can be likened to learning to fly an airplane while it is being built. Further- 
more, if you factor in the numerous competing agendas for school reform it is not 
surprising that the research agenda for educational technology is not particularly 
focused. Considering the progress that’s been made in the field in a very brief 
amount of time, we should not be overly critical of our shortcomings. With that 
said, though, I agree with the authors that as our field matures, and as external 
pressures mount, it makes sense to formulate a more focused research agenda that 
will better inform practice in our schools. 

The historical perspective that Roblyer and Knezek provide is very helpful for 
informing where we currently are as well as where we should be going. Al- 
though I tend to agree with Clark (1983, 1985, 1991, 1994, as cited in Roblyer 
& Knezek, 2003) in his criticism of media comparison studies as they were con- 
ducted, I accept Roblyer and Knezek’s premise that the goals of such research 
should be revisited. Perhaps part of the problem has been the proliferation of 
many small-scale studies that failed to acknowledge the range of variables in- 
volved in technology-based learning, and in effect, resulted in overly simplistic 
comparison of media. I agree with Kozma’s (1994) conclusions, published in a 
special issue on the topic, that we need to conduct research on “what ways can 
we use the capabilities of media to influence learning for particular students, 
tasks, and situations” (p. 18). That fits nicely with Roblyer and Knezek’s 
premise that we should “look at technologies not as delivery systems, but as 
components of solutions to educational problems” (p. 65). 

As the authors suggest, it makes sense to reach “some detente in the theory 
debates” (p. 64) and to proceed with an agenda that helps inform what works in 
actual instructional settings. Furthermore, it is reasonable to assert that technol- 
ogy-based methods should ultimately be able to demonstrate a “relative advan- 
tage” over other instructional methods. In this case, though, I would like to em- 
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phasize the word ultimately, and suggest that we investigate relative advantage 
in a balanced, forward-looking way. With the NCLB legislation and accompa- 
nying views of assessing student achievement, relative advantage for some could 
easily be reduced to quantitative measures. Although it’s clear that standardized 
tests and other objective measures are a force to be reckoned with and should be 
considered and thoughtfully employed, I would challenge the research commu- 
nity to continue exploring the relative advantage of transformative technology 
applications that may not fit as readily with the status quo. It seems clear that 
we need a balance of research that informs “what to do on Monday” with re- 
search that may ultimately inform what we do on Monday five or ten years 
hence. A broad interpretation of relative advantage seems critical to accommo- 
date interest in “established measures of education quality” as well as the trans- 
formative vision that Seymour Papert (1980) articulated with Logo two decades 
ago or what Alan Kay (2003) is exploring with Squeak today. 

The concept of relative advantage can also be applied to particular approaches 
within an instructional method. For example, Bernie Dodge (2003) recently 
suggested five models of motivation that could inform a research agenda inves- 
tigating approaches to the design and implementation of WebQuests. Although 
Dodge was not particularly interested in research that compares WebQuests to 
traditional approaches such as lectures, he did suggest that theory-based re- 
search was needed to help identify approaches within WebQuests that increase 
student motivation and learning. 

I appreciate Roblyer and Knezek’s suggestion that studies should seek to 
identify technology-based methods that “have the potential for unique and 
fairly consistent benefits in response to certain kinds of educational problems” 
(p. 65). Although this may be a matter of semantics, I prefer this language to 
the more prescriptive phrases of “best practice” and “scientifically-based evi- 
dence.” For me, such phrases tend to minimize the range and importance of 
variations in particular educational contexts. Teaching is both an art and a sci- 
ence, and as we seek to inform those elements that elevate the science of teach- 
ing, we should also seek to dispel the notion that effective teaching and learn- 
ing can be reduced to science alone. The authors did well to cite Berliner’s 
notion that “good science” requires not only trying to capture valid and reliable 
evidence, but also attempts to account for the complex set of conditions that 
led to improvement. 

GUIDELINES FOR DESIGN METHODS AND REPORTING 

With the caveat that research topics should remain sufficiently balanced to 
address a range of research questions, I support Roblyer and Knezek’s call for 
greater rigor within all approaches to research. It makes sense that research 
questions be stated in a way that the contributions of particular methods can be 
examined and tested, and that researchers seek to address the design shortcom- 
ing outlined. I agree that “they must be more comprehensive and informative 
about the methods and materials used, conditions under which studies take 
place, data sources and instruments, and subjects being studied; and they must 
emphasize coherence between their methods, findings, and conclusions” (p. 
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69). Although we clearly should refrain from simplistic comparisons of materi- 
als cited by Clark (1983, 1985, 1991, 1994, as cited in Roblyer & Knezek, 
2003), I also agree that research needs to help us sort through the myriad ap- 
proaches to using technology and help identify those “promising practices” that 
may consistently be effective in addressing particular educational needs. While 
qualitative studies can be particularly helpful in exploring new approaches and 
applications, as Berliner (2002) reminds us, good science requires detailed de- 
scriptions of the context and implementation strategies employed. Without in- 
cluding such descriptions in research reports, the transferability of such findings 
to other contexts is severely limited (Lincoln & Cuba, 1985). 

One goal of the proposed new millennium research agenda would be to in- 
form decision making in terms of the allocation of funds to support technol- 
ogy-based methods. This, of course, is reasonable, with one important caveat. 
Part of effectively implementing technology in schools depends upon identify- 
ing and establishing the essential conditions to support optimal implementa- 
tion. In many cases researchers have put forth conclusions about particular 
technology-based methods without adequately scrutinizing critical factors such 
as the adequacy of student access to the technology, the comfort level of the 
teacher, etc. I would argue that researchers must help identify the conditions 
necessary to realize the relative advantage of a technology-based method and 
avoid making premature conclusions about particular methods-based studies in 
which essential conditions are not in place. This supports Roblyer and Knezek’s 
notion that educational technology research needs to help discern fairly consis- 
tent benefits of particular technology-based methods. Without paying more at- 
tention to the necessary conditions, we will be much less likely to ascertain and 
document a method’s relative advantage. 

For example, Hall and Hord (2001) provide a model for studying change ef- 
forts that takes into account the concerns and levels of use of the implementers, 
as well as how the innovation is configured when implemented. They state that 
a major reason widespread change is often not successful is that change 
implementers do not fully understand what the innovation is and what it will 
look like when put into practice. Furthermore, they explain, “When a variety of 
configurations of an innovation are implemented, there is little likelihood that 
significant gains in student learning will be detected across all classrooms” (p. 
53). The researchers recommend creating an innovation configuration map that 
identifies major components of innovations and then describes observable varia- 
tions for each component. Such measures fit well with the need to define what 
effective technology integration looks like so that we might be able to discern 
positive learning outcomes and the relative advantage that may result. 

Providing this level of detail will enable researchers to obtain more valid and reli- 
able data to inform policy debates and classroom practice. A look at the well-publi- 
cized debate between Larry Cuban and Hank Becker may prove useful to this dis- 
cussion. As Roblyer and Knezek noted in their article, Cuban and colleagues have 
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articulated forceful arguments questioning the benefits derived from learning tech- 
nologies in schools (2001, as cited in Roblyer & Knezek, 2003). In response to 
Cuban's long standing arguments, based on his recent research, Becker wrote: 

Thus, in a certain sense Cuban is correct — computers have not trans- 
formed the teaching practices of the majority of teachers, particularly 
teachers of secondary academic subjects. However, under the right 
conditions — where teachers are perfectly comfortable and at least 
moderately skilled in using computers themselves, where the school’s 
daily class schedule permits allocating time for students to use com- 
puters as part of class assignments, where enough equipment is avail- 
able and convenient to permit computer activities to flow seamlessly 
alongside other learning tasks, and where teachers’ personal philoso- 
phies support a student-centered constructivist pedagogy that incor- 
porates collaborative projects defined partly by student interest — com- 
puters are clearly becoming a valuable and well-functioning 
instructional tool (2000, p. 29). 

I think Becker’s statement nicely illustrates the nature and complexity of edu- 
cational change with technology and points to the challenges that await re- 
searchers in the field. For those of us who have experienced the magic of teach- 
ing and learning with technology and “know” of its benefits, it is incumbent 
upon us to document those approaches to inform policy and classroom prac- 
tice. Toward that end, I applaud Roblyer and Knezek’s efforts to lay out some 
critical issues and put forth an agenda that will help us accomplish these goals. 
Clearly, there is much work to be done! 
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